Vendor diskspd instead of installing from chocolatey by KevinFairise2 · Pull Request #48688 · DataDog/datadog-agent

KevinFairise2 · 2026-03-31T16:08:55Z

What does this PR do?

Avoid relying on chocolatey, because it is a source of flakiness for those test. Go paged twice this week for that

Motivation

Describe how you validated your changes

Additional Notes

agent-platform-auto-pr · 2026-03-31T16:18:11Z

Gitlab CI Configuration Changes

Modified Jobs

workflow (configuration)

  workflow:
    rules:
    - if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
    - if: $CI_COMMIT_BRANCH == "main"
      variables:
        BAZEL_CACHE_POLICY_SUFFIX: -push
        GO_TEST_SKIP_FLAKE: 'false'
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
    - if: $DEPLOY_AGENT == "true" || $DDR_WORKFLOW_ID != null
    - if: $RUN_E2E_TESTS == "on"
    - if: $DEPLOY_INSTALLER == "true" || $DDR_WORKFLOW_ID != null
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
    - changes:
        compare_to: main
        paths:
        - '**/*_windows.go'
        - pkg/util/winutil/**/*
        - pkg/windowsdriver/**/*
        - pkg/util/pdhutil/**/*
        - pkg/util/crashreport/**/*
        - pkg/collector/corechecks/system/wincrashdetect/**/*
        - pkg/collector/corechecks/system/windowscertificate/**/*
        - pkg/collector/corechecks/system/winkmem/**/*
        - pkg/collector/corechecks/system/winproc/**/*
        - pkg/collector/corechecks/net/wlan/**/*
        - pkg/logs/launchers/windowsevent/**/*
        - pkg/logs/tailers/windowsevent/**/*
        - pkg/logs/util/windowsevent/**/*
        - pkg/network/driver/**/*
        - pkg/config/**/*
        - cmd/agent/**/*
        - cmd/process-agent/**/*
        - cmd/security-agent/**/*
        - cmd/otel-agent/**/*
        - cmd/privateactionrunner/**/*
        - cmd/system-probe/**/*
        - cmd/trace-agent/**/*
        - cmd/installer/**/*
        - cmd/systray/**/*
        - cmd/otel-agent/**/*
      variables:
        SKIP_WINDOWS: 'false'
    - changes:
        compare_to: main
        paths:
        - comp/systray/**/*
        - comp/updater/**/*
        - comp/checks/agentcrashdetect/**/*
        - comp/checks/windowseventlog/**/*
        - comp/checks/winregistry/**/*
        - comp/metadata/hostsysteminfo/**/*
        - comp/trace/etwtracer/**/*
        - comp/etw/**/*
        - comp/notableevents/**/*
        - comp/publishermetadatacache/**/*
        - comp/softwareinventory/**/*
        - tools/windows/**/*
        - .bazel*
        - MODULE.bazel*
        - omnibus/**/*
        - packages/**/*
        - deps/**/*
        - rtloader/**/*
        - bazel/**/*
        - Dockerfiles/agent/windows/**/*
        - Dockerfiles/agent/entrypoint.ps1
        - Dockerfiles/agent/entrypoint.d.windows/**/*
        - chocolatey/**/*
        - tasks/msi.py
        - tasks/winbuild.py
        - tasks/winbuildscripts/**/*
        - tasks/windows_resources.py
        - tasks/systray.py
        - release.json
        - .gitlab-ci.yml
        - .gitlab/**/*
      variables:
        SKIP_WINDOWS: 'false'
    - changes:
        compare_to: main
        paths:
        - test/new-e2e/tests/fleet/**/*
        - test/new-e2e/tests/installer/windows/**/*
        - test/new-e2e/tests/windows/**/*
        - test/new-e2e/tests/sysprobe-functional/**/*
+       - test/new-e2e/tests/process/**/*
      variables:
        SKIP_WINDOWS: 'false'
    - if: $CI_COMMIT_TAG == null
      variables:
        SKIP_WINDOWS: 'true'

.windows_path_3

  .windows_path_3:
  - test/new-e2e/tests/fleet/**/*
  - test/new-e2e/tests/installer/windows/**/*
  - test/new-e2e/tests/windows/**/*
  - test/new-e2e/tests/sysprobe-functional/**/*
+ - test/new-e2e/tests/process/**/*

Changes Summary

Removed	Modified	Added	Renamed
0	2	0	0

ℹ️ Diff available in the job log.

KevinFairise2 · 2026-03-31T16:18:59Z

/trigger-ci

gh-worker-devflow-routing-ef8351 · 2026-03-31T16:19:02Z

View all feedbacks in Devflow UI.

2026-03-31 16:19:01 UTC ℹ️ Start processing command /trigger-ci

2026-03-31 16:20:14 UTC ℹ️ Gitlab pipeline started

Started pipeline #105297794

agent-platform-auto-pr · 2026-03-31T16:42:52Z

Files inventory check summary

File checks results against ancestor da417011:

Results for datadog-agent_7.79.0~devel.git.303.040987a.pipeline.105426504-1_amd64.deb:

No change detected

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b0b702f747

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

wiyu

Thanks!

Ishirui

Woo reliability!
I wonder if we should start maintaining a list of all these artifacts we're vendoring on S3, where and why they are used. Could be useful if we ever need to update one for example.

aiuto · 2026-04-02T15:58:31Z

+	diskSpdArtifactKey = "processes/DiskSpd.zip"
+	// DiskSpdExe is the path to the amd64 diskspd executable after extraction
+	// (Microsoft DiskSpd release layout: amd64/diskspd.exe inside the zip).
+	DiskSpdExe = "C:/diskspd/amd64/diskspd.exe"


Is this part of the build images or are we going to actually vendor it into the source?
Or really, processes/DiskSpd.zip? Is that checked in.
My ask is only that if we are actually vendoring anything into the source tree, it lives in /third_party/...

That's not vendored in the source it is venodred in a dedicated S3 bucket we use for E2E tests third party dependencies.
https://datadoghq.atlassian.net/wiki/spaces/ADX/pages/5040342019/E2E+-+Use+a+third+party+artifact+in+test

cit-pr-commenter-54b7da · 2026-04-02T18:41:05Z

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: 31281b0b-8146-43c1-883f-072542af203c

Baseline: 3874f15
Comparison: 975f3f2
Diff

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	docker_containers_cpu	% cpu utilization	+0.03	[-2.95, +3.01]	1	Logs

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	otlp_ingest_logs	memory utilization	+0.70	[+0.60, +0.80]	1	Logs
➖	ddot_logs	memory utilization	+0.56	[+0.49, +0.62]	1	Logs
➖	quality_gate_idle	memory utilization	+0.42	[+0.36, +0.47]	1	Logs bounds checks dashboard
➖	ddot_metrics	memory utilization	+0.41	[+0.24, +0.59]	1	Logs
➖	uds_dogstatsd_20mb_12k_contexts_20_senders	memory utilization	+0.32	[+0.26, +0.38]	1	Logs
➖	docker_containers_memory	memory utilization	+0.29	[+0.21, +0.38]	1	Logs
➖	quality_gate_logs	% cpu utilization	+0.17	[-1.48, +1.82]	1	Logs bounds checks dashboard
➖	file_to_blackhole_1000ms_latency	egress throughput	+0.07	[-0.37, +0.50]	1	Logs
➖	file_to_blackhole_100ms_latency	egress throughput	+0.06	[-0.05, +0.18]	1	Logs
➖	docker_containers_cpu	% cpu utilization	+0.03	[-2.95, +3.01]	1	Logs
➖	uds_dogstatsd_to_api_v3	ingress throughput	+0.02	[-0.19, +0.23]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	+0.01	[-0.10, +0.11]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	-0.01	[-0.21, +0.20]	1	Logs
➖	file_to_blackhole_0ms_latency	egress throughput	-0.03	[-0.54, +0.48]	1	Logs
➖	file_to_blackhole_500ms_latency	egress throughput	-0.06	[-0.45, +0.34]	1	Logs
➖	quality_gate_metrics_logs	memory utilization	-0.11	[-0.35, +0.13]	1	Logs bounds checks dashboard
➖	file_tree	memory utilization	-0.14	[-0.20, -0.09]	1	Logs
➖	quality_gate_idle_all_features	memory utilization	-0.22	[-0.25, -0.18]	1	Logs bounds checks dashboard
➖	ddot_metrics_sum_cumulative	memory utilization	-0.23	[-0.37, -0.09]	1	Logs
➖	ddot_metrics_sum_delta	memory utilization	-0.23	[-0.41, -0.05]	1	Logs
➖	ddot_metrics_sum_cumulativetodelta_exporter	memory utilization	-0.24	[-0.46, -0.01]	1	Logs
➖	tcp_syslog_to_blackhole	ingress throughput	-0.24	[-0.43, -0.06]	1	Logs
➖	otlp_ingest_metrics	memory utilization	-0.46	[-0.62, -0.31]	1	Logs

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	observed_value	links
✅	docker_containers_cpu	simple_check_run	10/10	641 ≥ 26
✅	docker_containers_memory	memory_usage	10/10	273.52MiB ≤ 370MiB
✅	docker_containers_memory	simple_check_run	10/10	682 ≥ 26
✅	file_to_blackhole_0ms_latency	memory_usage	10/10	0.19GiB ≤ 1.20GiB
✅	file_to_blackhole_0ms_latency	missed_bytes	10/10	0B = 0B
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10	0.23GiB ≤ 1.20GiB
✅	file_to_blackhole_1000ms_latency	missed_bytes	10/10	0B = 0B
✅	file_to_blackhole_100ms_latency	memory_usage	10/10	0.19GiB ≤ 1.20GiB
✅	file_to_blackhole_100ms_latency	missed_bytes	10/10	0B = 0B
✅	file_to_blackhole_500ms_latency	memory_usage	10/10	0.22GiB ≤ 1.20GiB
✅	file_to_blackhole_500ms_latency	missed_bytes	10/10	0B = 0B
✅	quality_gate_idle	intake_connections	10/10	3 = 3	bounds checks dashboard
✅	quality_gate_idle	memory_usage	10/10	173.89MiB ≤ 181MiB	bounds checks dashboard
✅	quality_gate_idle_all_features	intake_connections	10/10	3 = 3	bounds checks dashboard
✅	quality_gate_idle_all_features	memory_usage	10/10	485.97MiB ≤ 550MiB	bounds checks dashboard
✅	quality_gate_logs	intake_connections	10/10	3 ≤ 6	bounds checks dashboard
✅	quality_gate_logs	memory_usage	10/10	203.28MiB ≤ 220MiB	bounds checks dashboard
✅	quality_gate_logs	missed_bytes	10/10	0B = 0B	bounds checks dashboard
✅	quality_gate_metrics_logs	cpu_usage	10/10	353.62 ≤ 2000	bounds checks dashboard
✅	quality_gate_metrics_logs	intake_connections	10/10	3 ≤ 6	bounds checks dashboard
✅	quality_gate_metrics_logs	memory_usage	10/10	427.72MiB ≤ 475MiB	bounds checks dashboard
✅	quality_gate_metrics_logs	missed_bytes	10/10	0B = 0B	bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.

### What does this PR do? Chocolatey was removed in: #48688 But a later PR update a test to use it again, hence it breaks with the conflict. Adds back chocolatey to buy some time properly fixing the test. #48833 ### Motivation ### Describe how you validated your changes ### Additional Notes

…lean up (#48833) # Changes - skips `TestLanguageDetectionWindows` since the e2e test is blocked by a gRPC stream cycling bug (connection + disconnection) after MSI reinstall ([s.UpdateEnv()](https://github.com/DataDog/datadog-agent/blob/1f8ef2cb695f5700a729dc4465657271fba5fef3/test/new-e2e/tests/process/windows_test.go#L314) does a full microsoft software install to "reset" the environment for a new test). This is an issue in the testing infrastructure which leads to the core agent being unable to pick up languages https://datadoghq.atlassian.net/browse/CXP-3410. I manually validated the language detection is still working. Again this is due to the testing setup. Running the python script ``` ❯ sshpass -p "<PASSWORD>" ssh -o StrictHostKeyChecking=no <SERVER> 'Set-Content -Path C:\sleep.py -Value "import time; time.sleep(600)"; Start-Process -FilePath "C:\Program Files\Datadog\Datadog Agent\embedded3\python.exe" -ArgumentList "C:\sleep.py" -WindowStyle Hidden; Start-Sleep 60; Get-Process python' ``` We can see the python language appear ``` ❯ sshpass -p "<PASSWORD>" ssh -o StrictHostKeyChecking=no <SERVER> '& "C:\Program Files\Datadog\Datadog Agent\bin\agent.exe" workload-list --json' | grep -o '"Language":{[^}]*}' | sort | uniq -c | sort -rn 96 "Language":{} 1 "Language":{"Name":"python"} ``` - removes chocolatey dependency — use the agent's embedded Python (`embedded3/python.exe`) instead of installing via chocolatey (flaky due to rate limits, see #48688) - makes `TestLanguageDetectionWindows` less flaky - removes PID matching and replaces with a check for any process with `language=python` instead of matching by PID - use persistent SSH session to start Python via `remoteHost.Start()` so it survives across SSH commands Co-authored-by: matthew.geng <matthew.geng@datadoghq.com>

Vendor diskspd instead of installing from chocolatey

b1fa2fc

dd-octo-sts bot added the internal Identify a non-fork PR label Mar 31, 2026

github-actions bot added the short review PR is simple enough to be reviewed quickly label Mar 31, 2026

dd-octo-sts bot added team/container-experiences team/agent-devx labels Mar 31, 2026

github-actions bot added long review PR is complex, plan time to review it and removed short review PR is simple enough to be reviewed quickly labels Mar 31, 2026

KevinFairise2 force-pushed the kfairise/vendor-diskpsd branch from 02e8de1 to 2025cc4 Compare March 31, 2026 16:14

github-actions bot added medium review PR review might take time and removed long review PR is complex, plan time to review it labels Mar 31, 2026

Trigger on process windows tests

b0b702f

KevinFairise2 force-pushed the kfairise/vendor-diskpsd branch from 2025cc4 to b0b702f Compare March 31, 2026 16:16

github-actions bot added short review PR is simple enough to be reviewed quickly and removed medium review PR review might take time labels Mar 31, 2026

KevinFairise2 marked this pull request as ready for review April 1, 2026 09:08

KevinFairise2 requested review from a team as code owners April 1, 2026 09:08

KevinFairise2 added changelog/no-changelog No changelog entry needed qa/no-code-change No code change in Agent code requiring validation labels Apr 1, 2026

chatgpt-codex-connector bot reviewed Apr 1, 2026

View reviewed changes

Comment thread .gitlab-ci.yml Outdated

Address codex comment

040987a

wiyu approved these changes Apr 1, 2026

View reviewed changes

Ishirui approved these changes Apr 2, 2026

View reviewed changes

aiuto reviewed Apr 2, 2026

View reviewed changes

github-actions bot added medium review PR review might take time and removed short review PR is simple enough to be reviewed quickly labels Apr 2, 2026

gh-worker-dd-mergequeue-cf854d bot merged commit 975f3f2 into main Apr 2, 2026
271 checks passed

gh-worker-dd-mergequeue-cf854d bot deleted the kfairise/vendor-diskpsd branch April 2, 2026 17:49

github-actions bot added this to the 7.79.0 milestone Apr 2, 2026

gengnamstyle mentioned this pull request Apr 3, 2026

[CXP-3401][agent] skip broken e2e windows language detection test + clean up #48833

Merged

KevinFairise2 mentioned this pull request Apr 3, 2026

Add back chocolatey ... #48836

Merged

Conversation

KevinFairise2 commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Describe how you validated your changes

Additional Notes

Uh oh!

agent-platform-auto-pr bot commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Gitlab CI Configuration Changes

Modified Jobs

Changes Summary

Uh oh!

KevinFairise2 commented Mar 31, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 bot commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

agent-platform-auto-pr bot commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Files inventory check summary

Results for datadog-agent_7.79.0~devel.git.303.040987a.pipeline.105426504-1_amd64.deb:

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

wiyu left a comment

Choose a reason for hiding this comment

Uh oh!

Ishirui left a comment

Choose a reason for hiding this comment

Uh oh!

aiuto Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KevinFairise2 Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cit-pr-commenter-54b7da bot commented Apr 2, 2026

Regression Detector

Regression Detector Results

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

CI Pass/Fail Decision

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

KevinFairise2 commented Mar 31, 2026 •

edited

Loading

agent-platform-auto-pr bot commented Mar 31, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 bot commented Mar 31, 2026 •

edited

Loading

agent-platform-auto-pr bot commented Mar 31, 2026 •

edited

Loading

aiuto Apr 2, 2026 •

edited

Loading